Goal-Oriented Sensitivity Analysis of Hyperparameters in Deep Learning

Tackling new machine learning problems with neural networks always means optimizing numerous hyperparameters that define their structure and strongly impact performances. In this work, we study the use of goal-oriented sensitivity analysis, based on Hilbert–Schmidt independence criterion (HSIC), for hyperparameter analysis optimization. Hyperparameters live in spaces are often complex awkward. They can be different natures (categorical, discrete, boolean, continuous), interact, have inter-dependencies. All makes it non-trivial to perform classical analysis. We alleviate these difficulties obtain a robust index is able quantify hyperparameters’ relative network’s final error. This valuable tool allows us better understand make optimization more interpretable. illustrate benefits knowledge context derive an HSIC-based algorithm apply MNIST Cifar, data sets, but also approximation Runge function Bateman equations solution, interest scientific learning. method yields both competitive cost-effective.